Numerical Relativity with AMReX

AMReX Community Breakout, HPSF Conference 2025

Miren Radia

University of Cambridge

May 7, 2025

Introduction

Who are we?

  • I am a Research Software Engineer (RSE) in Research Computing Services, University of Cambridge.
  • Previously, I completed a PhD in numerical relativity.
  • I am involved with several numerical relativity codes/projects/collaborations:

GRTeclyn

Katy Clough

Eugene Lim

Juliana Kwan

Paul Shellard

Pau Figueras

Ulrich Sperhake

MHDueT

Miguel Bezares

Carlos Palenzuela

UKNR CCP

Eugene Lim

Katy Clough

Mark Hannam

Geraint Pratten

Patricia Schmidt

What is Numerical Relativity (NR)?

  • Numerical Relativity is an area of research that uses numerical methods to find and investigate solution to Einstein’s equation of General Relativity (and other modified theories of gravity),

    \[ \underbrace{G_{ab}}_{\text{curvature}} + \underbrace{\Lambda g_{ab}}_{\text{cosmological constant}} = \underbrace{8\pi T_{ab}}_{\text{energy-momentum tensor}}. \]

  • We recast the equations into a set of coupled non-linear hyperbolic PDEs with elliptic constraints that we solve for the initial data.

  • Common problems include compact object binaries (black holes, neutron stars, more exotic objects) but NR is also used to study cosmological problems and perform “numerical experiments” for fundamental physics.

  • We are often interested in the gravitational waves (GWs) emitted in a simulation, particularly in the astrophysical compact binary case.

Common methods

  • Discretization:
    • Finite differences
      • Cell-centered
      • Vertex/node-centered
    • Finite volume (for fluids)
    • Spectral methods
    • Discontinuous Galerkin (DG) methods
  • Finite-difference mesh refinement:
    • Block-structured AMR
      • “Moving boxes” for compact object binaries
      • More adaptive for more exotic spacetimes
    • Oct-tree AMR
  • Method of lines
    • Fourth order Runge-Kutta time integration
    • Fourth or sixth-order spatial discretization
  • Kreiss-Oliger dissipation
  • Boundary conditions:
    • Outgoing radiation (Sommerfeld)
    • Symmetric/reflective
    • Extrapolating
    • Multipatch approaches: separate (e.g. spherical) outer grids

NR codes

There are many NR codes/frameworks including (but by no means limited to):

Oct-tree AMR

  • GRAthena++
  • Gmunu
  • Dendro-GR
  • GRAMSES

[Pseudo-]spectral/DG codes

  • SpEC/SpECTRE
  • bamps
  • SFINGE

Other

  • SENR/NRPy+

Block-structured AMR

  • Cactus/Einstein Toolkit\(^{\dagger}\)
    • McLachlan, LazEv, Maya, Lean, Canuda, …
  • BAM
  • PAMR/AMRD libraries
  • Hahndol/PARAMESH
  • HAD
  • CosmoGRaPH
  • Simflowny/MHDueT\(^{\dagger}\)
  • GRChombo/GRTeclyn\(^{\dagger}\)

\({\dagger}\) denotes codes/frameworks that are either transitioning to or developing support for using AMReX.

GRTeclyn

The GRTeclyn code

  • GRTeclyn is an in-development port of GRChombo (previously built on the Chombo libraries) to AMReX.
  • “Teclyn” is Welsh for “tool”.
  • Features
    • Amr based
    • Cell-centered but finite-difference rather than finite-volume
      • Fourth-order cell-centered interpolator from coarse to fine (CellQuartic)
      • Fourth-order interpolation from fine to coarse
    • Fourth-order Runge-Kutta time integration
    • Currently, just BinaryBH example which evolves black-hole binaries.
    • Designed to be flexible and easy to adapt to new problems.

Recent development

  • Black-hole puncture tracking.
    • Allows us to track the “centres” of black hole during the simulation
    • Uses AMReX’s particle infrastructure
    • Typically 2 BHs so just 2 puncture particles so maybe overkill
    • Good experience for future particle use
  • AMD MI300A hackathon
    • Lots of variables so high register pressure in RHS kernels
    • Legacy complicated data structure from GRChombo increases this
    • Refactoring led to a ~2x speedup in RHS kernels (~40% overall) on AMD MI300A APUs.
  • Currently working on adding support for basic matter.

MHDueT

The MHDueT code

  • MHDueT is a code that is automatically generated using the Simflowny infrastructure.
  • Previously used the SAMRAI library.
  • Features
    • AmrCore based.
    • Mostly node-centered
      • Fifth order polynomial interpolation from coarse to fine
    • Custom fourth order Runga-Kutta time integration
    • Supports Magnetohydrodynamics (MHD)
      • High resolution shock capturing with WENO reconstruction
      • Arbitrary EOS via HDF5 tabulated input
    • Can simulate binaries of boson stars, neutron stars, black holes and more.
    • Auto-generated using Simflowny infrastructure to allow minimal user programming.

Recent development

  • Addition of nodal syncing when filling boundary nodes to prevent instabilities (only appeared on GPUs).
  • Refactoring of custom higher-order extrapolating BCs to speed up these loops ~3000x on Nvidia A100 GPUs.
  • Reimplementation of reduction calculation to use AMReX ParReduce functions rather than custom code ported from SAMRAI which did not work on GPUs, nor was it thread-safe.
  • Dependency build scripts are now just Spack environments rather than a collection of custom shell scripts.

Future work

Particles

  • Extend “diagnostic” particles in GRTeclyn to allow high-order interpolation of state variables at arbitrary points in the domain for use with
    • GW extraction on coordinate spheres
    • Apparent horizon finding (~the boundary of black holes)
    • Matter flux around coordinate spheres
  • Add physical particles in GRTeclyn to model collisionless dark matter particles.
  • Add geodesic tracer particles to investigate behaviour around compact objects.
  • MHDueT already uses particles in their old SAMRAI version to model neutrino transport in high-density, hot fluid matter.
    • Port this feature to AMReX version.

UKNR Collaborative Computational Project

  • We have been awarded a grant to build a roadmap for the creation of a UK Numerical Relativity Collaborative Computational Project/Community (CCP) as part of the Computational Science Centre for Research Communities (CoSeC) programme.
  • This would provide long-term funding for our scientific community including shared cross-community code development.
  • As part of the technical branch of this grant we will:
    • Benchmark the performance of existing NR codes in use by UK researchers, and assess their readiness for future systems.
    • Identify opportunities to make data formats more interoperable and allow codes to share simulation/initial data.
    • Investigate the possibility of developing common [analysis] tools for NR codes.

Any questions?